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ABSTRACT: Engineering design is a complex process both for students to participate in and for 
instructors to assess. Informed designers use the key strategy of conducting experiments as they 
test ideas to inform next steps. Conversely, beginning designers experiment less, often with 
confounding variables. These behaviours are not easy to assess in educational settings because 
they occur throughout the design process. This paper reports on a two-fold study carried out to 
test the model for identifying student behaviours during design experimentation. The first phase 
uses the process data from 48 middle-school students designing an energy-plus house. The study 
utilized learner interaction data sets collected through automatic, unobtrusive logging of student 
actions in a CAD platform. The analysis of learner process data is compared to student 
performance on an open-ended post-test. The second phase correlates the number of 
experiments students conducted to the quality of student prototypes. The results suggest that 
the proposed model can be used to identify, characterize, and assess student strategies 
associated with conducting experiments. Implications of this work are relevant to engineering 
and design educators as well as researchers interested in the role of learning analytics in studying 
complex processes. 

Keywords: Experimentation, engineering design, K-12, learning analytics 


1 INTRODUCTION 

Learning analytics (LA) is a powerful approach to assessing complex learning processes in rich learning 
environments (Bienkowski, Feng, & Means, 2012; USDE, 2016). The use of learning analytics to study 
complex learning processes such as human problem solving is an opportunity to advance knowledge in 
ways that were not possible before (e.g., Worsley & Blikstein, 2014). When compared to traditional 
methods for studying and assessing student learning, technology-based approaches offer advantages by 
increasing personalization, accessibility, and efficiency (Pellegrino, 2002; USDE, 2016). As learners 
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engage with technology, they leave a trail of ways in which they use the technology (captured as 
interaction and process data), and the outcomes and the impact of using such technology (learning 
data). By integrating computational methods with learning sciences, learning analytics expand research 
capabilities in understanding the complex learning processes and nuances among learners. In this study, 
we specifically explore engineering design, a systematic and iterative process engineers use to solve ill- 
defined, ill-structured problems. 


When studying the student learning process, the open-ended nature of design is exactly what makes it 
difficult to assess. The qualitative approaches that have been explored rely on video-recorded design 
sessions, student written journals, or reports, but these approaches do not allow for capturing and 
characterizing students' own design processes on a large scale. This paper proposes a model to 
characterize and assess student strategies to conduct experiments as part of the engineering design 
process using learning analytics tools and techniques. This model, Experimentation Strategies in Design, 
is based on literature on student designers that suggests that novice designers perform fewer 
experiments as a part of their design process as compared to informed designers (Crismond & Adams, 
2012). However, as student designers progress in their practices, they perform experiments that range 
from confounding experiments to systematic controlled experiments. In this study, we hypothesize and 
test the Experimentation Strategies in Design model that characterizes the strategies students use to 
conduct experiments in a computer-aided design (CAD) environment as they design energy-plus or 
positive energy buildings. The CAD platform allows automatic, unobtrusive logging of student actions. 
The first phase of the study compares the clickstream data to student performance on an open-ended 
post-test. The second phase correlates the number of experiments to the quality of student prototypes. 

2 LITERATURE REVIEW 


2.1 Engineering Design 

Engineering design is an inquiry that can result in a process, a virtual product, or a physical product. 
Therefore, besides the direct support CAD tools provide to the design process, these tools can also be 
used for other purposes such as instruction and assessment (Xie, Zhang, Nourian, Pallant, & Hazzard, 
2014). Furthermore, when physical experimentation is required, some of these tools allow the designer 
to complete the cycle by 3-D printing the objects to create a prototype. 

Competency in engineering design is not easy to assess. The complexity of the design process and a 
designer's performance during this process can be effectively assessed neither solely through the 
performance of the designed product nor with a simple paper-pencil test. The open-ended nature of the 
design process may lead to different possible solutions by different students. Moreover, not only the 
final solution is important, but also the steps students go through as they approach the problem and 
identify an optimal solution. Several qualitative techniques have been explored as alternatives to assess 
the design process (e.g., Purzer, Goldstein, Adams, Xie, & Nourian, 2015; Worsley & Blikstein, 2014). 
Worsley and Blikstein performed a combined human-computer analysis of students as they completed a 
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short engineering design task. Their approach included the use of machine learning on human-coded 
video data to identify patterns in engineering design and developed a fine-grained representation of 
engineering practices. Their work is notable in the specific recommendations they provided to improve 
engineering education and discussions of how a hybrid human-computer analysis approach can make 
learning analytics more central to education research. 


As we examine experimentation behaviours during engineering design, it is necessary to define the 
place of scientific inquiry in relationship to design inquiry. Inquiry and engineering design are both 
dynamic processes (NAGB, 2013). Nonetheless, engineering design and scientific inquiry also have 
differences, specifically regarding the type of questions that guide the inquiry. In our approach, we see 
scientific inquiry and design inquiry to be integrated in ways that the designer seamlessly engages in 
both (Purzer et al., 2015). The design process starts with a problem or need that has many possible 
solutions. These solutions are tested and refined through micro-iterations until an ultimate solution is 
chosen. During this process, the designers must conduct valid experiments to learn about the variables 
and about how the different elements work in order to optimize the solution (Crismond & Adams, 
2012 ). 

2.2 Assessment of Engineering Design 

In prior studies, several approaches that combine qualitative data with quantitative analyses have been 
used to assess engineering design. For example, Atman and colleagues used verbal protocol analysis to 
identify differences between novice (student) and expert (professional) designers (Atman, Chimka, 
Bursic, & Nachtman, 1999; Atman et al., 2007). In these studies, researchers asked the participants to 
think aloud while working on a task about designing a playground. Although this technique has been 
demonstrated to be effective in capturing elements of engineering design, some verbalizations do not 
always represent the design knowledge that students applied (Atman, Kilgore, & McKenna, 2008). 
Furthermore, this data collection process is labour-intensive and time consuming. 

Another assessment approach is to request, as part of a design report, a description of how students 
applied the design process to solve a problem (Hirsch, Berliner-Heyman, Carpinelli, & Kimmel, 2012). 
Hirsch and colleagues (2012) asked one question prompting students to write about specific steps they 
used in their design process. In another effort to assess student design, Sims-Knight and collaborators 
(2004) used concept maps to assess student understanding of the engineering design process. In these 
examples, however, students' ability to describe the steps used to describe a design process does not 
imply that they know how to apply this knowledge. 

Students have also being asked to critique other students' design processes. For instance, Bailey (2008) 
implemented this strategy employing a scoring and feedback guide that students used to evaluate a 
given design solution. Hsu, Cardella, and Purzer (2014) followed a similar approach, asking elementary 
students to review another hypothetical student's design process and comment on 1) "what is good 
about the design process" and 2) "how the process can be improved" (p. 306). In this case, students 
were interviewed in order to explore their understanding about the design process. Additionally, Purzer, 
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Hilpert, and Wertz (2011) created a survey aimed at identifying the challenges students faced during a 
design project and how these challenges changed over time. Finally, Sobek (2002) and Seepersad, 
Green, & Schmidt (2006) introduced design journals as part of semester-long undergraduate courses. 
While the diversity of assessment methods is promising, the inferences that can be made with these 
results can also be diverse. 


Researchers have also used graphical software to capture the design rationale of a design team (van 
Schaik, Scanlan, Keane, Takeda, & Gorisse, 2011). During the design of an unmanned air vehicle, van 
Schaik and colleagues (2011) asked designers to record their rationales using Compendium, a 
collaborative mapping software. The findings of this study suggest that the software was useful in 
capturing the design rationale on certain phases of the design process. However, during the 
embodiment phase, when designers are making the connection between the concept and the final 
design, it was more difficult to capture the designers' rationale. A possible explanation is that during 
that phase, the designers are immersed in the CAD tool. Therefore, the authors suggested that these 
tools should be embedded within the CAD tool in order to be more effective. Furthermore, although this 
approach uses technology to organize the data, those ultimately responsible for reporting their design 
process are the designers themselves. 

These assessment methods discussed above involve manual approaches, which are time consuming and 
non-scalable (Xie, Zhang, Nourian, Pallant, & Hazzard, 2014). It would be infeasible to analyze the data 
from hundreds of students, gathered from think-aloud protocols, design documents, or interviews. 
Moreover, all these techniques have demonstrated certain limitations, and there is yet not a perfect 
recipe to assess engineering design practices and related competencies. 

2.3 Technology-Based Assessments 

Engineering design has been included within one of the three assessment targets of the Technology and 
Engineering Literacy (TEL) Framework for the 2014 National Assessment of Educational Progress (NAGB, 
2013) and in the winter of 2014 a computer-based NAEP TEL test was taken by a national sample of 
eighth-grade students in the United States (NAEP, 2014). Technology-based assessments have now 
acquired high relevance because of the advantages they can provide. Large amounts of relevant data, 
real-time feedback, and scalable and personalized support can be achieved now with the use of these 
type of assessments (USDE, 2010). Still, taking advantage of automated assessment techniques becomes 
challenging for open-ended tasks such as engineering design (Worsley & Blikstein, 2014). Therefore, 
Worsley and Blikstein (2014) suggest that it is necessary to combine learning analytics with human- 
based qualitative analysis to be able to draw strong conclusions about the learning process. 

Document analysis using computational methods is one of the techniques that have been explored to 
characterize student design processes (Dong, Hill, & Agogino, 2004). Dong et al. (2004) used latent 
semantic analysis (LSA) to analyze collaborative team design documentation and personal reflections on 
the process. Another technique is the use of learning analytics to assess different elements of the 
learning process (e.g., Romero-Zaldivar, Pardo, Burgos, & Delgado, 2012; Xie, Zhang, Nourian, Pallant, & 
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Hazzard, 2014). This technique uses the interactions between learners and technological devices to draw 
inferences about the learner, the tool, and the learning process (Mirriahi, Gasevic, Long, & Dawson, 
2014). Through unobtrusive logging of user interactions, hundreds of megabytes or even gigabytes are 
generated to be analyzed as a form of assessment (Xie, Zhang, Nourian, Pallant, & Bailey, 2014). 


There are several ways of logging and analyzing user interactions. For example, Blikstein (2011) and 
Piech, Sahami, Koller, Cooper, and Blikstein, (2012) employed screenshots of user interactions to assess 
different processes. Worsley and Blikstein (2014) also used snapshots, but in this case, to assess the 
engineering design process. They first qualitatively analyzed the snapshots to create codes to the actions 
performed by the participants. After that, they used an algorithm to characterize the student design 
processes. Finally, they corroborated their findings and provided additional details through a qualitative 
analysis. Using the same activity, Worsley and Blikstein (2013) also captured and analyzed gesture data 
using Kinect to compare the differences between novice and expert hand movements. 


CAD tools have also been used to understand the design process (e.g., Sung, Ritchie, Rea, & Corney, 
2011; Xie, Zhang, Nourian, Pallant, & Bailey, 2014; Xie, Zhang, Nourian, Pallant, & Hazzard, 2014). Purzer 
and collaborators (2015) used a CAD tool in an exploratory study to understand how engineering design 
and scientific explanations related to each other. The learning activity employed by the research team 
consisted of applying the engineering design process to create a model of an urban city block of 
buildings to minimize the energy needed in those buildings. Students tested their designs, analyzed 
building performance, and refined their designs after considering constraints and trade-offs. The CAD 
tool had the capability of recording and reproducing the student design processes via time-lapse 
photography. These videos were qualitatively analyzed and contrasted to student models and 
reflections. 


The same CAD tool has been employed to log and analyze the design process in order to characterize 
students' design pathways. For example, Xie, Zhang, Nourian, Pallant, & Hazzard (2014) used the Time 
Series Analysis method to characterize differences in students design processes. The interactions with 
the software were analyzed as a sequence of data points using techniques such as autocorrelation 
analysis or cross-correlation analysis (Wei, 2005). These techniques helped to identify recurrent actions 
and sub processes, as well as the amount of time spent on these activities. From two different studies, 
the research team concluded that valuable insights can be drawn from this type of analysis (Xie, Zhang, 
Nourian, Pallant, & Bailey, 2014; Xie, Zhang, Nourian, Pallant, & Hazzard, 2014) because: 

• unobtrusive logging allows the teacher or the research team to gather data from the 
interactions while the students focus on the design challenge 

• student engagement in the design challenges can be easily assessed from the data 

• the scalable nature of automated data analysis allows the teacher to have a real-time 
assessment of the situation 

• the macro- and micro-level design iterations can be expressed and visualized with a visual model 

• differences between design processes for different populations can be detected 
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• the learning analytics can be complimented with qualitative data for additional insights and a 
deeper exploration of the design process 

• Time Series Analysis can be used to identify the effect of instructional strategies in an 
engineering design challenge 


Technology-based assessment techniques seem to be a promising tool to scale-up the evaluation of 
student learning. However, additional work is required before these automated assessment approaches 
do not require manually analyzed data to make a deeper sense of the design process. 

2.4 Understanding Differences between Designers 

An important distinction to make in order to assess the design process effectively is to understand what 
can be considered an informed designer versus a beginning designer. In this research, we are 
particularly interested in student designers at the pre-college level. The lower anchor of the student 
grouping is "beginning designers" or student designers with little to no design experience. The high end 
of this student development scale is the "informed designer" (Crismond & Adams, 2012). Because K-12 
and undergraduate students are not likely to accumulate the level of practice needed to attain expert 
designer status, their skills when developed are less reliable and their understandings are much more 
situation-specific than those of experts (Crismond & Adams, 2012). Thus, informed designers, while not 
experts, are engaged and knowledgeable in design beyond that of true beginners. 

Researchers have described and contrasted the characteristics of expert and novice designers from 
different perspectives (Atman et al., 2007; Crismond & Adams, 2012; Worsley & Blikstein, 2014). In the 
context of engineering design, it is usually expected that expert designers spend more time planning and 
building than novice designers (Worsley & Blikstein, 2014). Specifically, expert designers spend more 
time in problem scoping and gathering significantly more information than novice designers (Atman et 
al., 1999; 2007). Problem scoping can be described as the phase in which the designers understand the 
problem, gather information, identify criteria, and state assumptions (Atman et al., 2007). The time 
spent on this phase is positively correlated to customer satisfaction and the quality of the design (Jain & 
Sobek 2006). 

Worsley & Blikstein (2014) identified that novice designers undo their actions more often than experts. 
Atman et al. (1999) highlighted a similar result, suggesting that senior designers make more transitions 
between different phases than novice designers. Additional qualitative analysis showed that experts set 
short-term goals, as they would go back to planning activities after creating certain artifacts (Worsley & 
Blikstein, 2014). 

Crismond and Adams (2012) analyzed more than 80 publications on design cognition, and created a 
framework to integrate the findings in order to support design teaching and learning. The framework 
consists of a matrix describing how beginning designer practices differ from informed designer practices 
(Crismond & Adams, 2012). While the beginning designer is assumed to have little to no experience or 
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formal training in design, the informed designer has certain experience and formal training in design, 
and is located in-between the beginning designer and the expert designer. The framework includes nine 
critical design practices (Crismond & Adams, 2012): 1) understand the challenge, 2) build knowledge, 3) 
generate ideas, 4) represent Ideas, 5) weigh options and make decisions, 6) conduct experiments, 7) 
troubleshoot, 8) revise/iterate, and 9) reflect on the process. Among these design strategies, the 
informed designers iterate through these steps by generating and evaluating ideas, conducting 
experiments, reviewing literature, and thinking critically. 


Regarding experimentation strategies, informed designers identify and conduct valid tests to assess the 
appropriateness of the proposed solution (Crismond & Adams, 2012). These strategies inform further 
actions on the design process aiming to optimize the final product. Hence, experimentation of informed 
designers focuses on relevant variables that will allow them to find an optimal solution. In contrast, the 
beginning designers conduct very limited experimentation. The experimentation of beginning designers 
often confounds the variables by manipulating multiple variables at the same time. Therefore, results of 
the experimentation process are not clear or informative to the process. 


The next section describes a model that intends to support the identification, characterization, and 
assessment of student strategies to conduct experiments within the engineering design process. 

3 PROPOSED MODEL: EXPERIMENTATION STRATEGIES IN DESIGN 


Experimentation is an important element of engineering design (Crismond & Adams, 2012; Litzinger, 
Lattuca, Hadgraft, & Newstetter, 2011). However, experimentation strategies in the context of 
engineering design are different from those in science (Schauble, Klopfer, & Raghavan, 1991). 
Experimentation in science aims to understand the relationship among variables, to explain scientific 
phenomena. Experimentation in design focuses on identifying and manipulating the relevant variables 
to optimize the outcome. 

Experimentation in an engineering learning environment can take different forms. For instance, students 
can collect data and confirm design performance. Students can also use experimentation to compare 
two alternatives of a product or design. This experimentation allows them to make informed decisions 
among the potential alternatives. Moreover, students can modify independent variables and use the 
collected data to inform future design decisions. 

We describe the differences on experimentation strategies between a beginning and an informed 
designer as a continuum (see Figure 1). Supporting evidence for our Experimentation Strategies in 
Design Model comes from prior literature by Crismond and Adams (2012), Schauble et al. (1991), and 
Litzinger et al. (2011). At the beginning of this range, a beginning designer tends to run few or no tests 
on a design prototype because they may feel conducting experiments is not relevant for the design 
activity. Thus, the beginning designer focuses on creating a prototype without collecting any data. 
Furthermore, when beginning designers collect data, they often fail to use these results to inform 
further design actions. As the student designer gains experience, she or he is expected to realize how 
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important data collection is in order to understand what is happening with their prototype. However, 
these initial experiments tend to be unfocused and the designer continues modifying multiple variables 
within a single experiment, which Crismond & Adams (2012) calls a confounded experiment. Therefore, 
the designer is unable to identify the relevant variables and optimize the solution. 

The informed designers are located towards the end of the continuum. With experience, the designer 
knows that it is important to focus on one specific variable at a time and to perform valid tests in order 
to inform the next steps towards a comprehensive solution. Hence, the designer carries out specific 
systematic experiments followed by specific actions related to the outcomes of the experiment. 




Few experiments with 
confounded variables 


Multiple experiments 
with confounded 
variables 


Focused, systematic 
experiments 


Figure 1. Characteristics of experimentation strategies along a continuum of level of experience. 


In this context, we define a confounding experiment as the process of collecting data and modifying 
multiple independent variables before collecting data again, leading to ambiguous results. Likewise, we 
define a systematic experiment as the process of collecting data, modifying only one variable, and 
collecting data again to visualize the effect of this independent variable on the dependent variable given 
by the data collection. These extremes of the continuum represent the changes in experimentation 
strategies as a student moves towards informed design. 


Figure 2 describes the sequence of generic steps, based on investigate-and-redesign task sequence 
(Crismond, 2001), which comprises systematic experiments. In the first step, the designer creates a 
prototype to evaluate. The second step consists of collecting and analyzing data focusing on specific 
independent and dependent variables. Based on the results from the data analysis, the designer could 
make some changes to the prototype modifying the independent variables identified as relevant to 
optimize the performance of the prototype. During the last step, the designer goes back to collect and 
analyze more data to identify the effect of the changes made. 
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1. Build / Modify a 
prototype 



* 1 

r -n 

4. Collect and analyze data 
on specific variables 


2. Collect and analyze data 
on specific variables 

<_ 4 



% 


3. Modify these 
variables on the 
prototype 




How many different variables is the designer considering on 
different experiments? 


Figure 2. Sequence of steps for systematic experiments. 


Our model of Experimentation Strategies in Design aims to identify a sequence of actions with a CAD 
tool to describe student experimentation strategies with large datasets. Our model can also describe 
students' approaches to experimentation. Questions such as how many experiments a student carries 
out, how many of these experiments are systematic, and how many times a student iterates through a 
specific experiment, can help identify how experienced a designer is. 

4 METHODS 


The purpose of this study is two-fold. We first propose a learning analytics model to describe student 
experimentation processes. We then utilize this model to correlate student experimentation strategies 
with the performance of their designed prototypes. In the sections below, we first introduce the 
computer-aided design CAD tool students used, called Energy3D. We then describe particulars of the 
research methods performed. 

Research Question 1: What is the relationship between the number of systematic experiments and 
levels of student strategic knowledge? 


Research Question 2: What is the relationship between the number of systematic experiments and the 
quality of student design solutions? 

4.1 Energy3D 

Energy3D is an open-source software (Concord, 2015) that allows students to create 3D buildings and 
simulate energy consumption and construction costs (Xie, Zhang, Nourian, Pallant, & McIntyre, 2014). 
Energy3D provides the tools for a novice to design, analyze, and build energy-efficient buildings 
(Concord, 2015). The software includes energy consumption simulations based on computational 
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physics to allow the user to make informed design decisions. Figure 3 shows three examples of buildings 
and simulations students created using Energy3D. 



Figure 3. Sample buildings created using Energy3D (Concord, 2015). 

Energy3D logs all user interaction data in the background in a non-intrusive way as students build and 
test their solutions. Energy3D creates a text file for each student with a list of all the interactions 
including the following information: 1) the date and time the action was carried out; 2) the file in which 
the action was carried out; 3) the description of the action (e.g., Add, Edit, Move, Resize); and 4) the 
object towards which the action was directed. Energy3D also has the capability of reproducing the 
design process as a running slide show (design replays), similar to time-lapse photography. Not only is 
the model recreated, but student reflections are also reproduced synchronously. Thus, design replays 
allow the educator or researcher to understand what students do during the design process (Purzer et 
al., 2015). 

4.2 Participants 

This study was conducted at a large, urban middle school in the Midwest United States. Participants 
included 199 students across eight (8) eighth-grade classes (ages 13-14) during the fall of 2014. The 
students who took part in the study had little to no previous design experience. The sample included a 
nearly even mix of female (n = 98) as male (n = 101) students. The number of participants was 48 for the 
first part of the study and 55 for the second part of the study. After a brief introduction to Energy3D (as 
students had no previous experience with the software or other CAD software), students were given the 
task of designing an energy-plus home, a home that consumes no net energy over a year, along with 
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other design constraints such as cost. (See the complete challenge in Appendix A.) 


In addition, the task explicitly asked students to design three types of houses: Colonial, Cape Cod, and 
Ranch. Instructors allocated two class periods for each of the three types of houses for a total of six days 
of design time. Over these six days, students were instructed to plan, construct, experiment, analyze, 
optimize, and document each of their three designs. Students were reminded to revisit any of their 
designs over the duration of the six days, allowing multiple iteration opportunities. 

4.3 Characterization of “Conducting Experiments” Actions 

Experimentation behaviours may vary among different design challenges, tools, and contexts. Therefore, 
our identification of student experimentation strategies started by defining what systematic 
experiments a student can carry out for this particular context and within the boundaries of the design 
challenge outlined earlier. Using a preliminary qualitative data analysis on student reports and 
reflections for a similar design challenge, the research team identified a set of experiments and variables 
that students are likely to explore while solving the design challenge. As shown in Table 1, we explain 
student experimentation strategies by a set of design actions, design variables these actions have 
explored, and design analysis tools used to test these variables. For example, in the scenario that a 
student is exploring how trees planted near a building impact its energy performance. The student 
reflects, "the amount of windows with trees relatively close to the windows allows the sunlight to warm 
the house in the window, but the trees provide shade for the windows in the summer months which ends 
up cooling the house." To accomplish the given task, the student would need to interact with Energy3D 
to create, move, or resize windows or trees. Then, they would use the analysis tools to identify what 
could be the best approach based on their constraints. The effect of these changes on the annual energy 
consumption is not the same for different seasons (e.g., summer and winter). Hence, students would 
need to collect data more than once while modifying these variables in order to optimize their solution. 


Table 1. Design goals, actions, variables, and analysis tools using Energy3D 


Experimentation Goal 

Design Actions 

Design 

Variables 

Design Analysis 

Tools 

Testing changes in energy 
consumption 

Add, move, or resize windows 

and move trees to increase 

energy efficiency at different 

seasons 

Windows and 

Trees 

Energy Annual 
Analysis/Modify the 

Season 

Conducting experiments to 
optimize annual consumption 
of energy 

Modify the roof shape or the 
solar panels to increase the 
energy generated by solar 
panels 

Roof-Solar 

Panel 

Interactions 

Energy Annual 
Analysis/Modify the 

Season 


The experimentation strategies reported by the students through reflections in the context of an 
Energy3D design challenge were employed to contextualize this study within the proposed framework 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 


301 






JOURNAL OF LEARNING ANALYTICS 


S • LAR 

SOCIETY for LEARNING 
ANALYTICS RESEARCH 

(2016). Using learning analytics to characterize student experimentation strategies in engineering design. Journal of Learning Analytics, 3(3), 
291-317. http://dx.doi.org/10.18608/jla.2016.33.14 

represented in Figure 1 as continuum of design experience and in Figure 2 as a sequence of design 
actions. The purpose of this process is to identify such sequences of interactions through the clickstream 
data so that this information can be used to assess experimentation strategies. As summarized in Table 
1, student experiments are evaluated as two distinctly different experimentation activities: 1) by testing 
changes in energy consumption after adding or modifying windows or trees, and changing between 
seasons and 2) by conducting experiments to optimize the annual consumption of energy based on 
modifications in shape, location, and orientation of roofs and solar panels. 

4.3.1 Experimentation Strategy #1: Testing changes in energy consumption after adding or 
modifying windows or trees, and changing between seasons. 


This design behaviour represents experiments that students carry out by adding, modifying, or removing 
windows or trees, and collecting energy consumption data. Students explore the interaction between 
trees and windows to reduce temperature during the summer season while increasing temperature in 
the winter. The location of the windows and trees as well as the type of trees (i.e., deciduous vs. 
evergreen) could keep the house very cold during the winter, generating an additional consumption due 
to heating. Here, the analysis tools such as the "Energy Annual Analysis" and "Modify Seasons" are 
critical in supporting decisions. Figure 4 depicts the sequence of interactions for this experimentation 
process. As described earlier, we expect students to 1) identify one variable, which in this case would be 
the location or size of a window or tree; 2) collect data: energy annual analysis or change in solar 
radiation given by the month of the year; 3) modify the independent variable; and 4) collect data again 
to test their modifications. 


1. Add, move or 
resize windows 
/trees 



4. Energy 
Annual Analysis 
/ Seasonal 
^ Change 


2. Energy 
Annual Analysis 
/ Seasonal 
c Change j 



3. Add, move or 
resize windows 
/ trees 



Figure 4. Sequence of actions for a student testing changes in energy consumption after adding or 
modifying windows or trees, and changing between seasons. 
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4.3.2 Experiment Strategy #2: Conducting experiments to optimize the annual consumption of 
energy based on modifications in shape, location, and orientation of roofs and solar 
panels. 

This design behaviour represents conducting experiments to optimize the interaction between the roof 
shape and the solar panel location and angle. Depending on the location of the house and the 
orientation of the solar panels, the energy efficiency can change. This process was also identified from 
the qualitative analysis of student reports and reflections after being exposed to the design challenge. 
For example, one of the students stated in their designer notes: 


I reconfigured the solar panels' angle and placement on the house many times. I tested several 
different angles for the roof on the side with solar panels to determine which angle generated 
the most energy, then moved the panels around on the properly angled roof to maximize the 
energy generation. 


Figure 5 depicts the sequence of steps the students are expected to follow when they focus on these 
variables for systematic experimentation. According to our model, Experimentation Strategies in Design, 
the sequence of design behaviour includes 1) add/modify solar panels or roofs, 2) energy annual 
analysis, 3) add/modify solar panels or roofs, and 4) another energy annual analysis (see Figure 5). 

4.4 Data Collection and Data Analysis 

4.4.1 Capturing Learner Interaction with Clickstream Data 

The data analysis explaining learner interactions was conducted using the clickstream data (Figure 6). 
First, the clickstream data are captured in a text file for each student participating on the design 
challenge. Once the text file has been loaded, the different actions were organized into multiple time 
series (i.e., one per type of action) describing how many times this action occurred during a particular 
minute within the activity. Within the Energy3D platform, there are two main types of actions: 
construction and analysis. Construction actions correspond to adding, modifying, removing, resizing, or 
moving features such as walls, windows, doors, and roofs of the house. Analysis actions correspond to 
executing data collection tools such as the energy annual analysis or changing a variable to see the 
effect of such modification on energy performance. 
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3. Add/Modify 
Solar Panels or 
Roofs 


Figure 5. Sequence of actions for a student conducting experiments to optimize the annual 
consumption of energy based on modifications in shape, location, and orientation of roofs and solar 

panels. 


Using the proposed model, we identified the number of data collection activities with potential 
confounding variables, and the number of systematic experiments. To distinguish among beginning-to- 
informed designers, the number of data collection activities related to annual energy consumption are 
recorded as a measure of experimentation with potential confounded variables to represent the early 
levels of design proficiency shown in Figure 1. To distinguish informed designers, the number of 
sequences representing controlled systematic experiments were identified and counted. In our context, 
controlled systematic experimentation has been defined as completing two separate interactions with 
trees, windows, and analyses, as well as a reflection on the experiment outcomes. All these activities are 
expected to occur within a window of time. A calibration study was conducted to determine how long 
this window of time should be. This calibration process determined that the average time that students 
needed to run the analysis, analyze the results, and make changes in their design, and execute the 
analysis again to complete the cycle of systematic experiments was 20 minutes. 
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Figure 6. Data analysis process for the clickstream data. 
Note: The gray arrows point to sample files. 


4.4.2 Post-test of Design Strategies 

After students concluded the design challenge, they completed a post-test that measured students' 
design strategies, specifically students' ability to demonstrate their design decisions and justify these 
decisions with scientific reasoning. The test consisted of four open-ended questions asking about 
strategies students used to keep a house warm during the winter or cool during the summer, improving 
overall home energy efficiency, and tests students could conduct in efforts to improve energy efficiency: 

1. Let's say we want to keep this building warm in the winter. What are some strengths of this 
design that would help? What are some aspects of this design that will make it hard to keep 
warm in winter? 

2. If we wanted to keep this building cool in the summer, what are some strengths of this design 
that would help? What are some aspects of this design that will make it hard to keep cool in 
summer? 

3. Let's say you are an engineer and someone asked you to improve this building so it would be 
more energy efficient. What are some ideas you might try? 

4. What kinds of tests or experiments would you try to see if your ideas were working? What kinds 
of information would be useful or helpful? 

A scoring protocol was used to evaluate the student responses to the open-ended questions. Responses 
were grouped into three categories of design strategies demonstrated (i.e., high, intermediate, and low) 
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as depicted in Table 2. The protocol for scoring the post-tests was developed specific to the strategic 
knowledge expected to be gained through the Energy3D system and to quantify student responses in 
order to be able to compare post-test data to the quantitative log data statistically. 


Table 2. Levels of Strategic Design Knowledge 


Level 

Description 

Example 

A - High 

Student suggests experimentation within 
the system showing evidence of 
engagement in using the Energy3D 
software. The variables student has noted 
are reasonable and are related to the 
software parameters. 

"Adding more solar panels and moving 
the trees to the front would make this 
building cooler in the summer. It would 
block the sun , making it easier to get 
shade." 

B - Intermediate 

There is evidence of awareness of 
parameters/variables within the system. 
However, the elaboration of these 
variables is weak or incorrect. 

"Some ideas 1 might try could be putting 
more or less windows." 

C - Low 

There is no evidence of awareness of 
parameters/variables within the system. 

All ideas appear to be drawn from 
everyday experiences. 

"It would be hard to keep it warm in the 
winter if you have your windows open." 


Hypothesis 1 

The research team hypothesized that significant differences exist on the number of systematic 
experiments for the different levels of strategic knowledge. To evaluate this hypothesis, responses to 
the post-test instrument from 48 students were analyzed and compared to the experimentation 
strategies identified by the analysis of clickstream data. 

Hypothesis 2 

The second hypothesis is that a significant strong correlation exists between the number of systematic 
experiments and the quality of the design the students submitted. A product with a high quality should 
minimize the cost and the energy consumption while maximizing the area of the house. 

4.4.3 Calculation of Solution Quality 

We calculated design quality for a design solution with a formula that normalizes the key design 
parameters of cost, energy consumption, and area by dividing each parameter by the maximum value 
within the sample. Because the design criteria specified a maximum construction cost (i.e., $60,000) this 
value was used in normalizing variable costs. The quality for the design was calculated as: 
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Q x 


^ (( i - 

maxG4) \ \ 


Ex 

maxCf) 



In the quality formula, A represents the area of the building; max(x) represents maximum value for 
variable x in the sample; E represents annual energy consumption; C represents cost; and C/ /m represents 
budget limit for the challenge. The research team used the second design submitted by the students to 
test the hypothesis for two reasons. First, students typically spent the initial building design to explore 
the tool and get familiar with it. Second, some students did not have the chance to spend enough time 
working on the third design. Therefore, the second design was expected to be the most thoughtful, in 
which they considered all criteria and constraints. The average quality score was 0.46 with a standard 
deviation of 0.15. The lowest quality score was zero while the highest score was 1.01. 


5 RESULTS 


Hypothesis 1: The number of systematic experiments is significantly different for different levels of 
student strategic design knowledge. 

Student experiments were grouped based on whether they were focused and systematic, nonexistent, 
or confounded. The maximum total number of confounded (non-systematic) experiments by a single 
student was 66, with a mean of 12.52 and a standard deviation of 14.65. The number of students who 
collected data within the Energy3D system ten times or more was 23; seven students did not collect any 
annual energy consumption data. 


Table 3. Descriptive Statistics of the Number of Systematic and Non-Systematic Experiments 


Experimentation 

Min 

Max 

Mean 

Standard 

Median 

Type 




deviation 


Systematic 

0 

33 

3.93 

7.02 

0.5 

Non-systematic 

0 

66 

12.52 

14.65 

6.5 


The maximum number of systematic experiments by an individual student was 33. Six students out of 48 
completed more than 10 systematic experiments while 23 students did not do any systematic 
experiments. The average number of experiments was 3.93 with a standard deviation of 7.02. 

Figure 7 shows a comparison between two students. Each plot has tree graphs that represent the 
number of specific actions over time. For example, the left side of Figures 7a and 7b shows: 1) number 
of construction actions over window objects; 2) number of construction actions over tree objects; and 3) 
number of energy annual analysis actions. Figure 7a depicts actions from a student who carried out 
multiple energy annual analysis while focusing on windows and trees (see minute 40). This student also 
focused their attention over roof and solar panels, around minute 75. Figure 7b shows a student that 
carried out a limited number of experiments and that were not closely related to specific actions. 

Forty-eight students completed the post-test. Responses from 18 students were identified as high level, 
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15 students as intermediate level, and 14 students as low level. Table 4 presents the descriptive 
statistics on the number of systematic experiments and total number of experiments for these three 
groups. The number of experiments and number of systematic experiments did not satisfy the 
assumption of normality for the analysis of variance. Therefore, the data was transformed using the 
square root transformation, which is an appropriate approach for a count variable (McDonald, 2007). 


Number of actions per minute 


Number of actions per minute 


JL J_ JlaJ\ 

lL 

A AA A 

1 111 1 III 



0 20 40 60 80 100 

Time 



Time 


(a) Student doing systematic experiments. 

Number of actions per minute Number of actions per minute 




(b) Student doing random experiments without focusing on specific variables. 

Figure 7. Number of student actions over time, related to energy annual analysis, windows, and trees. 


Analysis of variance showed a significant difference among the groups, both for the number of 
systematic experiments (F(2, 45)= 9.429, p<0.001), and for the total number of experiments (F(2, 45)= 
12.42, p<0.001). A post-hoc analysis showed that the students with a low-level response did a 
significantly smaller number of systematic experiments than students with intermediate-level response 
(difference between the means of square roots= -2.14, p<0.001) and those with high-level response 
(difference between the means of square roots= -1.40, p=0.01). For the total number of experiments, 
again the group with low-level response showed a significantly smaller number than the group of 
intermediate-level responses (difference = -2.95, p<0.001) and the group of high-level responses 
(difference = -2.03, p=0.002). However, the relationship between students with intermediate and high 
levels of strategic knowledge was not significant for systematic experimentation or number of 
experiments (difference between the means of square roots = 0.74, p=0.26, difference between the 
means of square roots = 0.9, p=0.25, respectively). 
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Table 4. Descriptive Statistics for the Number of Systematic Experiments and All Experiments 



Systematic Experiments 

All Experiments 


Mean 

Standard Deviation 

Mean 

Standard Deviation 

High Level 

4.21 

4.81 

13.58 

4.81 

Intermediate Level 

8.53 

10.68 

23.87 

21.16 

Low Level 

0.29 

0.83 

4.14 

5.49 


Hypothesis 2: The number of experiments is positively correlated to the quality of the prototype 
submitted by the students. 

Fifty-five students who submitted final design solutions were included in this analysis. The analysis 
indicated a positive correlation between the number of experiments performed (systematic and non- 
systematic) and the product quality. Pearson correlations were performed to analyze the relationships 
among the number of controlled systematic experiments, the number of all experiments, and the 
product quality. An r value of less than 0.1 is considered a weak correlation; a moderate correlation 
ranges from 0.25 to 0.45, and a strong correlation is 0.5 or higher. The results indicated a moderate 
correlation both between the number of all experiments and the final product quality (r= 0.37, p=0.005), 
and between the number of systematic experiments and the final product quality (r= 0.38, p=0.004). In 
addition, the result for the correlational analysis suggested that students who performed the majority of 
systematic experiments also collected data multiple times (r= 0.92, p <0.001). 

6 DISCUSSION AND CONCLUSIONS 

This study proposed a model to identify, characterize, and assess student experimentation strategies 
during engineering design: the Experimentation Strategies in Design Model. We have proposed a 
continuum of design proficiency that starts with having no experience with engineering design to 
becoming informed designers. In this model, beginning designers (those at the beginning of the 
continuum) do not consider experimentation as an important strategy and only collect data few times in 
a non-strategic way (Crismond & Adams, 2012). As the students gain experience, they start to conduct 
more experiments, which become more systematic through exposure and experience. The systematic 
experiments focus on specific variables and iterations among data collection procedures and 
modification of these variables to optimize the end product (Schauble et al., 1991). 

The Experimentation Strategies in Design Model helps to identify both the number of experiments and 
the number of systematic experiments a student would complete from learning analytics data. The 
research team first hypothesized that there would be a significantly different number of experiments as 
well as a significantly different number of systematic experiments based on the level of strategic 
knowledge depicted by student performance on the post-test. The results suggest that there is a 
significant difference between students with a low level of design strategies and students with 
intermediate and high levels. Other studies have noted similar statistical differences between students 
exhibiting low, medium, and high levels of informed design behaviours, where highly and moderately 
reflective students had higher gains in informed design compared to those with low reflectivity 
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(Goldstein, Purzer, Adams, & Xie, 2015). Our results provide evidence supporting the first part of the 
continuum that describes the differences between beginner and informed designers' experimentation 
strategies. However, we found no significant differences for the overall number of experiments nor 
systematic experiments between intermediate and high levels of student strategic design knowledge. A 
possible explanation is that students, regardless of their level of strategic design knowledge are still 
novice designers and are apt to design "depth-first" (Cross, 2000), meaning they fully develop a single or 
few ideas rather than experimenting and learning from multiple design possibilities regardless of their 
strategic design knowledge. Because the students were exposed to these activities for a limited time, 
exposure to the relevant variables for systematic experimentation were also limited. However, results 
also emphasize the importance of a range of design practices and strategies in conjunction with knowing 
when and how to use them (Adams, Turns, & Atman, 2003). It is understandable that students who 
exhibit greater degrees of strategic approaches to design also exhibit a greater degree of systematic 
experimentation. 

The second hypothesis intended to supplement the validation of the model by comparing the quality of 
a prototype with the quality of their design. The research team used students' second prototypes to test 
this hypothesis. The results suggest that there is a moderate correlation between the number of 
experiments and the prototype quality score (r=0.37, p=0.005). There is also a moderate correlation 
between the number of systematic experiments and the quality score (r=0.38, p=0.004). Not 
surprisingly, there was a very strong correlation between the overall number of experiments and the 
number of systematic experiments. These findings suggest that students who did more general 
experimentation also did more systematic experiments. Additionally, this greater quantity and quality of 
experimentation is related to having a better quality design, as described by the Experimentation 
Strategies in Design Model. 

In future research, our Experimentation Strategies in Design Model can be further explored and applied 
to assess student experimentation strategies in other contexts where students need to make controlled 
experiments. The next steps include the identification of different types of systematic experiments, how 
they can affect student learning outcomes, and the quality of their design solutions. A follow-up study 
may also explore the effect of providing scaffolding to students on experimentation strategies (e.g., 
Seah, Vieira, Magana, & Dasgupta, 2016). The model can also be used to assess the change on 
experimentation strategies after instruction. 

7 LIMITATIONS AND FUTURE WORK 

The main limitation of this study corresponds to the specificity of the context in which the proposed 
model was implemented. The middle-school engineering challenge using Energy3D has specific variables 
that the students should consider for the experiments to be performed (e.g., energy consumption, solar 
radiation). The group of students and the alternative available actions limited the number of levels we 
could find within the continuum proposed by the framework. Furthermore, the validation of the model 
was only given by external instruments, such as the open-ended questions and the quality product 
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design. Additional research is required to capture students' rationale for their experiments and compare 
it to what is identified by this model. A possible approach for this future study could be using think- 
aloud protocols, in which students describe their actions while working on the design challenge (Seah et 
al., 2016). Another limitation involves the use of aggregated data per minute. Aggregating the data 
removes detail in some cases about which of two actions occurred first within that period. In the future, 
this could be improved by treating the data in a non-aggregated way, so we could identify the specific 
order of actions. A third limitation of this study is the design quality score. Assessing design with a 
quantitative score is difficult, especially when the number of design options is nearly unlimited. The 
current approach attempts to be mindful of repeatability, subjectivity, and credibility, by considering the 
main variables the design challenge focuses on (i.e., energy consumption and cost, as related to the area 
of the house). Collecting additional design data will allow us to take a systematic approach to 
understanding how well our design quality score addresses those concerns. 
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APPENDIX A 


Energy-Plus Home Design 


An energy-plus house, over the course of a year, produces more renewable energy than the energy it 
consumes. A client wants to build such an eco-friendly house in Boston. Your job is to come up with 
three designs using Energy3D, following the engineering design cycle illustrated below. 

An Engineering Design Cycle 

Before you do anything, you should 
read this document carefully to make 
sure that you fully understand the 
design specs, which specify what the 
client wants. Then you should think 
about what you will do to meet those 
specs. You can discuss with your 
classmates about ideas. After you 
have some basic ideas, you can start 
with constructing a house in 
Energy3D that you think may appeal 
to the client. Once you are happy with it, you can analyze its energy performance. Energy3D can 
automatically calculate the annual energy usage for heating and cooling. Based on your analysis of the 
results, you will decide how to revise your design to improve the energy efficiency of the house. 
Meanwhile, your design must meet all the other specs such as style, size, and material cost, as described 
below. 



Design Specs 


The client wants the energy-plus house to be in one of the following three styles: 



Colonial 

Cape Cod 

Ranch 

Budget <$60,000, area 120-160 m 2 . 

Budget <$60,000, area 100-150 m 2 . 

Budget <$60,000, area 150-200 m 2 . 

height 8-10 m 

height 7-9 m 

height 4-6 m 
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Note that the budget only covers material costs (labour cost is not considered). The house must also 
meet the following requirements: 

• Each side of the house must have at least one window. 

• Tree trunks must be at least two meters away from the walls of the house (i.e., the distance 
must be greater than the length of two cells of the blue ground grid when it appears). 

• Do NOT add entry porches, dormers, chimneys, garages, or driveways. 

• Do NOT add more than 40 solar panels (regardless of their conversion efficiency). 

• There is no need to design any interior structure such as rooms, floors, or stairs. 

• Roof overhang must be less than 50 centimeters wide (the default is 25 centimeters). 

• Doors cannot be wider than two meters or taller than three meters. 

• Keep the room temperature of the house to be 20°C all the time. 


Since you cannot predict what the client may like, you will design a Colonial house, a Cape Code house, 
and a Ranch house. You will spend two class periods planning, constructing, experimenting, analyzing, 
optimizing, and documenting each of the three designs. At the end of this project, you will summarize 
the pros and cons of each design based on its aesthetics, material cost, and energy efficiency and 
present your best design to the class. Until this project is over, you can always revisit and improve a 
design. 


Instruction 


The client has provided the initial computer models of the land parcel on which the house will be 
constructed. These models also come with existing surrounding trees, grayed out because you are not 
allowed to move or cut them. The files for these models, colonial.ng3, capecod.ng3, and ranch.ng3, can 
be found in your USB drive and can be opened from within Energy3D (using File > Open). The trees and 
land are identical in all the three files. The differences are in their specifications of area and height. 
Remember, don't start from scratch — you must always start with these provided files. 

To get started, 

1) Write your name on the tag attached to your USB drive. 

2) Connect the USB drive to the computer and then open the USB folder on the computer. 

3) Double-click energy3d.jar on the USB drive to run Energy3D. 

4) After Energy3D is ready, use the "File > Open" menu to open an ng3 file on the USB drive. You 
can choose any of the three files as your first design. 

5) Start working on your design. While designing, use the "Show Note" button on the task bar of 
Energy3D to open a text area below the 3D view window. The text area provides some 
instructions, questions, and a checklist to guide you through the design process. You must 
answer those questions based on your design. Save everything you have done in this file — do 
not save your work into a different file unless you want to keep a backup copy. 
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6) At the end of each session, remember to save your design. You MUST save all your work in the 
USB drive (as you may use a different computer next time). At the end of a class period, do NOT 

just pull the USB drive — use the EJECT function of the computer to safely remove it. 

7) Repeat for your second and third designs. 


Important Notes 

• You can only build one house on the platform. Don't put multiple houses on a single platform. 
(A house is defined as a space enclosed by one and only one set of connected walls.) 

• The house can be rotated and its height can be adjusted as a whole. Focus on its shape design 
initially and keep in mind that you can always raise, lower, or rotate it later. 

Always document your work and thoughts in the Note Area, especially when you are analyzing the 
energy efficiency of your design. Your notes will be saved when you save your design. 
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